Statistical Decision Making for Optimal Budget Allocation in Crowd Labeling
نویسندگان
چکیده
منابع مشابه
Statistical decision making for optimal budget allocation in crowd labeling
It has become increasingly popular to obtain machine learning labels through commercial crowdsourcing services. The crowdsourcing workers or annotators are paid for each label they provide, but the task requester usually has only a limited amount of the budget. Since the data instances have different levels of labeling difficulty and the workers have different reliability for the labeling task,...
متن کاملStatistical Decision Making for Budget Allocation in Crowdsourcing
In this short paper, we briefly describe some recent progress on statistical decision making for budget allocation in crowdsourcing. We address the budget allocation problem for two important labeling tasks in crowdsourcing: the categorization labeling task and pairwise ranking aggregation. We also show the connections between our work and the “proactive learning” framework proposed by Jaime Ca...
متن کاملData mining for decision making in engineering optimal design
Often in modeling the engineering optimization design problems, the value of objective function(s) is not clearly defined in terms of design variables. Instead it is obtained by some numerical analysis such as FE structural analysis, fluid mechanic analysis, and thermodynamic analysis, etc. Yet, the numerical analyses are considerably time consuming to obtain the final value of objective functi...
متن کاملOptimal Budget Allocation for Sample Average Approximation
The sample average approximation approach to solving stochastic programs induces a sampling error, caused by replacing an expectation by a sample average, as well as an optimization error due to approximating the solution of the resulting sample average problem. We obtain estimators of an optimal solution and the optimal value of the original stochastic program after executing a finite number o...
متن کاملContinuous Evolution of Statistical Estimators for Optimal Decision-Making
In many everyday situations, humans must make precise decisions in the presence of uncertain sensory information. For example, when asked to combine information from multiple sources we often assign greater weight to the more reliable information. It has been proposed that statistical-optimality often observed in human perception and decision-making requires that humans have access to the uncer...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: SSRN Electronic Journal
سال: 2014
ISSN: 1556-5068
DOI: 10.2139/ssrn.2408163